Skip to content

Bump data substrate to payload reservation (tx_service#578); add checkpoint-cycle multi-node test - #507

Open
liangjchen wants to merge 1 commit into
mainfrom
bump/data-substrate-payload-reservation
Open

liangjchen wants to merge 1 commit into
mainfrom
bump/data-substrate-payload-reservation

Conversation

@liangjchen

@liangjchen liangjchen commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

EloqDoc needs no source change for the bump.

What the bump brings

Details are in eloqdata/tx_service#578. In short:

  • A write-lock acquisition reserves its payload on the destination shard before the lock attempt, and reserved bytes count as used shard memory.
  • A request parked for shard memory is rejected with OUT_OF_MEMORY after shard_memory_wait_ckpt_cycles concluded checkpoint cycles (default 5). Before, most requests waited indefinitely. Log replay and standby forwarding are exempt.
  • New metric checkpoint_cycles_concluded_total.

Visible to EloqDoc users: under sustained memory pressure a write or read can now fail with an out-of-memory error where it used to wait.

The new test

ckpt_cycles.py reads each node's checkpoint_cycles_concluded_total on a live cluster and checks two cases that unit tests cannot stage:

  • Merged rounds (local and cloud mode). A data sync task is held at its start by a fault point. Rounds that start meanwhile merge into it and must not conclude; that node's count stays flat while the other nodes' counts advance. When the task is released, the count rises by one for the held round plus one per merged round.
  • Abandoned round (cloud mode). A voter leads a killed node's group; its round over that group is held after it joined the cycle while the group returns to its node. When released, the round abandons the group and the cycle still concludes, exactly once.

Like the other scripts in that directory, it is run by hand against a server built with fault points; it is not registered in CI.

Test plan

Run on 2026-11-13 inside the CI image, on EloqDoc main plus this PR's two files, with the data substrate at the #578 change (RelWithDebInfo, EloqStore data store, RocksDB log state, fault injection on):

  • cmake --build <dir> --target eloqdoc eloqdoc-tests host_manager
  • ctest --test-dir <dir> --label-exclude '^integration$': passed
  • Runtime targets eloqdoc-server-smoke, eloqdoc-ddl-recovery, eloqdoc-cluster-gate, eloqdoc-change-stream-drivers, eloqdoc-cdc-consumer: all passed
  • Multi-node, EloqStore local mode: smoke, dss_starting, replay_rerequest, ckpt_cycles, cdc_step_down, cluster_gate_multinode, ddl_split_recovery: 7 of 7 passed
  • Multi-node, EloqStore cloud mode: voter_takeover plus the same 7: 7 of 8 passed. ddl_split_recovery hung in its last scenario and passed on a rerun. The cause is a race that exists on main and is fixed separately in Fix a recovered range split dropped at replay finish: read leader and candidate terms as one pair tx_service#579; it is not related to this change.

Not run: the js suites, and any performance comparison.

The data substrate tree that was tested differs from the pinned commit by one re-indented line.

Before merging

The submodule points at the head of the #578 branch. tx_service merges by squash, so once #578 is merged this pointer must be moved to the resulting commit on tx_service main. Please do not merge this PR before that.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation
    • Added guidance on cloud-mode voter takeover, leadership handback, and checkpoint-cycle test scenarios.
  • Tests
    • Added multinode checks for checkpoint cycles when data-sync tasks are delayed or a node leaves and rejoins its group.
    • Added options to select scenarios and configure test runs, with failures reported across selected scenarios.

…i-node test

Moves the data substrate submodule to eloqdata/tx_service#578: write-lock
acquisitions reserve their payload on the destination shard before the lock
attempt, and a request parked for shard memory is rejected with
OUT_OF_MEMORY after a configured number of concluded checkpoint cycles.
EloqDoc needs no source change for it.

Adds cmake/tests/multinode/ckpt_cycles.py, which checks the per-node count
of concluded checkpoint cycles on a live cluster through the
checkpoint_cycles_concluded_total metric: rounds merged into a held flush
conclude only when it lands, one each, and a round whose node group changes
leader mid-round still concludes (cloud mode).

The README lists the new test, and voter_takeover.py, which it had left
out.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Adds a multinode test runner for checkpoint-cycle counts. It covers deduplicated data-sync rounds and cloud-mode voter takeover, documents the scenarios, and updates the data substrate subproject reference.

Changes

Multinode checkpoint-cycle tests

Layer / File(s) Summary
Metric controls and test documentation
cmake/tests/multinode/ckpt_cycles.py, cmake/tests/multinode/README.md
Adds helpers to read and check concluded-cycle counts, detect held fault points, and identify a specific startup race. Documents the checkpoint-cycle scenarios and command-line usage.
Deduplicated data-sync scenario
cmake/tests/multinode/ckpt_cycles.py
Holds a data-sync task while timer rounds accumulate, checks the resulting cycle count and stability, and verifies inserted documents can be read.
Cloud-mode takeover and runner
cmake/tests/multinode/ckpt_cycles.py, cmake/tests/multinode/README.md
Checks a held voter round when a node returns and reclaims its group. Registers scenarios and adds CLI options, cloud-mode skipping, and failure reporting.

Data substrate reference

Layer / File(s) Summary
Subproject reference update
src/mongo/db/modules/eloq/data_substrate
Updates the recorded data substrate commit reference.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Other

Merge Risk: Low · up to 38b3b

This updates the data substrate reference and adds manual multi-node checkpoint-cycle tests. EloqDoc already translates the new out-of-memory outcome. Before merging, move the submodule pointer to the commit produced when tx_service#578 is squash-merged into main, as the author plans.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names two changes in the changeset: the data substrate submodule bump to tx_service#578 (payload reservation) and the new multi-node checkpoint-cycle test (ckpt_cycles.py). Both are real par…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names two changes in the changeset: the data substrate submodule bump to tx_service#578 (payload reservation) and the new multi-node checkpoint-cycle test (ckpt_cycles.py). Both are real parts of the PR, and the submodule bump is the main functional change. The title is long (97 characters) but it is specific and not misleading. It does not mention the README update or the behavior change (out-of-memory rejection after concluded checkpoint cycles), which is acceptable because the title does not need to cover every detail.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A rabbit checks the rounds go by,
While held sync tasks wait nearby.
The voter counts, the node comes home,
And reads confirm the data’s shown.
Then off the rabbit hops to roam.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7c0bed83-fe03-43e1-87ca-dd647fc42fe2
Commits

Reviewing files that changed from the base of the PR and between 0413fc4 and 38b3b49.

Files selected for processing: 3

  • cmake/tests/multinode/README.md
  • cmake/tests/multinode/ckpt_cycles.py
  • src/mongo/db/modules/eloq/data_substrate

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant